Fixes #21: merge test coverage with e2e tests in CI#22
Conversation
metafates
left a comment
metafates
left a comment
There was a problem hiding this comment.
Thank you! Left some review comments
| if: runner.os != 'Windows' | ||
| run: | | ||
| go test -race -v -coverprofile=coverage.out -coverpkg=./... ./... | ||
| make coverage-ci TEST_FLAGS="-race -v" |
There was a problem hiding this comment.
Let's inline this so that workflow won't depend on Makefile
| go tool cover -func coverage.out | ||
|
|
||
| # get test coverage-ci | ||
| .PHONY: coverage-ci |
There was a problem hiding this comment.
.PHONY isn't needed here because we have MAKEFLAGS += --always-make
| coverage-ci: | ||
| go test $(TEST_FLAGS) -coverprofile=coverage.unit.out -coverpkg=./... ./... | ||
| go test $(TEST_FLAGS) -tags e2e -coverprofile=coverage.e2e.out -coverpkg=./... ./... || true | ||
| go run github.com/dlespiau/covertool@latest merge -o coverage.out coverage.unit.out coverage.e2e.out |
There was a problem hiding this comment.
If possible, I think we should avoid relying on third-party tools.
I think we can simplify this a bit.
Overall, it should look like this:
go test ./...- run unit testsgo test -tags e2e -count 1 ./examples_test.go- run e2e testsgo test -tags example -coverprofile coverage.out -coverpkg ./... ./... || true- run both unit tests and examples (which are invoked from e2e).
There was a problem hiding this comment.
I suggest two options:
- Simple, but it requires running unit tests twice (first without coverage, then again during the general test run)
# Run unit tests separately to catch errors
go test -race -v ./...
# Run all tests together for coverage (ignoring failures)
go test -race -v -tags e2e -coverprofile=coverage.out -coverpkg=./... ./... || true
go tool cover -func=coverage.out
go tool cover -html=coverage.out -o coverage/coverage.html- Using
go tool covdata. This is the official Go tool introduced in v1.20, designed exactly for this use case.
mkdir -p coverage/unit coverage/e2e coverage/merged
# Run unit tests and save raw coverage data to coverage/unit
go test -race -v -coverpkg=./... ./... -test.gocoverdir=coverage/unit
# Run e2e tests and save raw coverage data to coverage/e2e (ignoring failures)
go test -race -v -tags e2e -coverpkg=./... ./... -test.gocoverdir=coverage/e2e || true
# Merge data from both directories into coverage/merged
go tool covdata merge -i=coverage/unit,coverage/e2e -o=coverage/merged
# Convert merged data to text format (coverage.out)
go tool covdata textfmt -i=coverage/merged -o=coverage.out
# Generate a report to the console
go tool cover -func=coverage.out
# Save the report to text and HTML files
go tool cover -func=coverage.out -o coverage/coverage.out
go tool cover -html=coverage.out -o coverage/coverage.htmlWhich approach do you think is more suitable?
There was a problem hiding this comment.
Simple, but it requires running unit tests twice
I think it's good enough. Let's stick to this one.
But important note, though - running e2e won't affect the coverage, as it runs examples through separate process. We need to run those examples directly with example tag from examples/ dir with coverage.
There was a problem hiding this comment.
Great, then we can run it like this:
go test -race -v -tags e2e,example -coverprofile=coverage.out -coverpkg=./... ./... || true
2 participants
Fixes #21: merge test coverage with e2e tests in CI